feat(sleep): add correctness-first annotation case study - #3
feat(sleep): add correctness-first annotation case study#3viranovskaya wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The timing output currently mislabels what is being measured for object search and there is an incorrect type hint, both of which should be corrected before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new correctness-first sleep annotation case study to demonstrate that HED search can retrieve parallel sleep-stage annotations by meaning while preserving source identity, using a compact synthetic fixture and deterministic outputs. This advances the repository's goal of establishing shared benchmark patterns before a universal benchmark JSON contract is finalized.
Changes:
- Introduces a new
use_cases/sleep/case with a synthetic BOAS-modeled fixture, strict fixture validation, and independent numeric-label oracles to verify Basic/String/Object HED search correctness. - Adds deterministic correctness JSON + a generated Markdown report, along with an optional timing artifact that records local environment metadata.
- Adds pytest coverage and updates repository-level documentation to list and describe the new working sleep case.
File summaries
| File | Description |
|---|---|
| use_cases/sleep/src/sleep_case.py | Implements fixture validation, independent oracles, correctness gating across search engines, and optional timing output. |
| use_cases/sleep/src/report.py | Renders a deterministic Markdown report from the correctness JSON. |
| use_cases/sleep/README.md | Documents the scientific question, correctness contract, and how to run the case. |
| use_cases/sleep/json_specifications/README.md | Captures case-derived JSON field observations to inform future shared schema design. |
| use_cases/sleep/example/test_data/sleep_annotation_events.tsv | Adds the 14-row synthetic sleep annotation fixture table. |
| use_cases/sleep/example/test_data/sleep_annotation_events.json | Adds the sidecar mapping stage codes to HED annotations for each source stream. |
| use_cases/sleep/example/test_data_results/reports/sleep_correctness_report.md | Commits an example generated correctness report for the fixture. |
| use_cases/sleep/example/test_data_results/output/sleep_correctness.json | Commits deterministic correctness output for the fixture. |
| use_cases/sleep/example/test_data_results/figures/README.md | Tracks the committed figures directory with a placeholder README. |
| tests/test_sleep_case.py | Adds regression tests for correctness, determinism, and fixture contract enforcement. |
| README.md | Updates the case study list and example commands to include the new sleep case. |
| docs/use_cases/sleep.md | Updates docs to reflect that the sleep case is now working and describes its scope. |
Review details
- Files reviewed: 12/12 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| }, | ||
| { | ||
| "query_id": case["id"], | ||
| "engine": "Object search (parse and search)", |
There was a problem hiding this comment.
Good catch. The timed callable constructs each event HedString and performs the search, while QueryHandler is intentionally initialized outside the timed section. I renamed the label to Object search (event parsing and search) to make that boundary explicit, and added a regression test for the timing labels.
| path.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8", newline="\n") | ||
|
|
||
|
|
||
| def _timing_summary(function, runs: int) -> dict[str, float]: |
There was a problem hiding this comment.
Updated the return annotation to dict[str, int | float], since runs is an integer while the timing summaries are floats. The timing-contract test now also checks that runs remains an integer.
Why
Before a shared benchmark contract is designed, the repository needs a small correctness-first case showing that HED search can retrieve parallel sleep annotations by meaning while preserving their source. The case should also make clear where semantic retrieval ends and ordinary sleep analysis begins.
What changed
No annotation stream is treated as ground truth. This PR follows the current per-case repository structure and exposes a callable
run_case()that a future shared CLI can wrap; it does not define the universal benchmark JSON contract.Verification
Relates to #1.